Papers by Lakshya A Agrawal
LangProBe: a Language Program Benchmark (2025.findings-emnlp)
Copied to clipboard
Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia
| Challenge: | Composing language models into multi-step language programs is a mainstream paradigm for building AI systems, but tradeoffs in this space have only scarcely been studied before. |
| Approach: | They propose a benchmarking tool to evaluate the architectures and optimization strategies for language programs . they find that optimized language programs offer strong cost-quality Pareto improvement . |
| Outcome: | The proposed framework evaluates the impact of program architectures and optimizers on quality and cost. |